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(54) High precision sub-pixel spatial alignment of digital images 



(57) A high precision sub-pixel spatial alignment 
of digital images, one from a reference video signal 
and another from a corresponding test video signal, 
uses an iterative process and incorporates spatial 
resampling along with basic correlation and estimation 
of fractional pixel shift. The corresponding images 
from the reference and test video signals are captured 
and a test block is overlaid on them at the same 
locations to include texture from the images. FFTs are 
performed within the test block in each image, and the 
FFTs are cross-correlated to develop a peak value 
representing a shift position between the images. A 
curve is fitted to the peak and neighboring values to 
find the nearest integer pixel shift position. The test 
block is shifted in the test image by the integer pixel 
shift position, and the FFT in the test image is 
repeated and correlated with the FFT from the reference 
image. The curve fitting is repeated to obtain a 
fractional pixel shift position value that is combined 
with the integer pixel shift value to update the test 
block position again in the test image. The steps are 



repeated until an end condition is achieved, at which 
point the value of the pixel shift position for the 
test block in the test image relative to the reference 
image is used to align the two images with high 
precision sub-pixel accuracy. 
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Description 

BACKGROUND OF THE INVENTION 

5 [0001] The present invention relates to the matching of digital images, and more particularly to high precision 
sub-pixel spatial alignment of digital images. 

[0002] Accurate spatial alignment of digital images is of fundamental importance to many applications. One such 
application is the determination of picture quality using an instrument, such as the PQA200 Picture Quality Analyzer 
manufactured by Tektronix, Inc. of Wilsonville, Oregon, USA, where images from a reference video signal are compared 
with corresponding images from a test video signal that is a processed version of the reference video signal. The 
10 better the spatial alignment between a reference image and a corresponding test image, the more accurate the 
determination of the amount of degradation of the test video signal due to the processing. 

[0003] At pages 480-484 of Computer Image Processing and Recognition (1979) published by Academic Press, 
Ernest L. Hall describes a basic correlator method for template matching, which may be used for image alignment. 
Likewise J.J. Pearson et al in an article entitled Video-Rate Image Correlation Processor (IOCC 1977) published in SPIE 
Vol. 119 Application of Digital Image Processing, pages 197-205, describe a variant of the basic correlator method 

15 that uses phase information. For digital images with integer pixel shifts these correlation methods may be very 
accurate. However for a broader class of shifts, which involves both integer and fractional pixel shifts, the 
precision of the sub-pixel alignment becomes a very serious matter. Pearson et al describe a method for estimating 
the sub-pixel shifts by quadratic interpolation using the correlation values in the neighborhood of a peak. However, 
since the quadratic interpolation function does not necessarily match the true image shifting function, the accuracy 

2Q obtained by the quadratic interpolation is limited. 

[0004] What is desired is a high precision sub-pixel spatial alignment of digital images. 

BRIEF SUMMARY OF THE INVENTION 

[0005] Accordingly the present invention provides for high precision sub-pixel spatial alignment of digital 
25 images using an iteration method and spatial resampling. A high precision sub-pixel spatial alignment of digital 
images, one form a reference video signal and another from a corresponding test video signal, uses an iterative 
process and incorporates spatial resampling along with basic correlation and estimation of fractional pixel shift. 
The corresponding images from the reference and test video signals are captured and a test block is overlaid on them 
at the same locations to include texture from the images. FFTs are performed within the test block in each image, 
and the FFTs are cross-correlated to develop a peak value representing a shift position between the images. A curve 
30 is fitted to the peak and neighboring values to find the nearest integer pixel shift position. The test block is 
shifted in the test image by the integer pixel shift position, and the FFT in the test image is repeated and 
correlated with the FFT from the reference image. The curve fitting is repeated to obtain a fractional pixel shift 
position value that is combined with the integer pixel shift value to update the test block position again in the 
test image. The steps are repeated until an end condition is achieved, at which point the value of the pixel shift 
position for the test block in the test image relative to the reference image is used to align the two images with 
35 high precision sub-pixel accuracy. 

[0006] The objects, advantages and other novel features of the present invention are apparent from the following 
detailed description when read in conjunction with the appended claims and attached drawing. 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 

40 

[0007] Fig. 1 is a block diagram view of a system for high precision sub-pixel spatial alignment of digital 
images according to the present invention. 

[0008] Fig. 2 is an illustrative view of a comparison of an image from a reference video signal and an image 
from a corresponding test video signal to be spatially aligned with high precision sub-pixel accuracy according to the 
present invention. 

45 [0009] Fig. 3 is a flow diagram view of a high precision sub-pixel spatial alignment algorithm according to the 
present invention. 

[0010] Fig. 4 is an illustrative view of a cross-correlation surface showing a peak location for the high 
precision sub-pixel spatial alignment algorithm according to the present invention. 

50 DETAILED DESCRIPTION OF THE INVENTION 

[0011] Referring now to Fig. 1 an original, reference video signal from a source 10 is input to a video processing 
network 12. The video processing network 12 may include one or more compression/decompression passes. The 
output from the video processing network 12 is input to a decoder or similar device 14 to provide a source of a test 
video signal. The reference and test video signals are input to a video capture module 16 that captures corresponding 
55 images or frames from the two video signals. The corresponding image pairs from the video capture module 16 are 
input to a high precision sub-pixel spatial alignment detect module 18 to determine a position shift between the images 
of the reference video signal and the images of the test video signal. The position shift from the spatial alignment 
detect module 18 is input to a position shift module 20, which may be either in the reference or test signal path, to 
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precisely align the image pairs with sub-pixel accuracy from the video capture module 16 for further processing, such 
as by a picture quality analyzer 22. 

[0012] In Fig. 2 the reference and test images 24, 26 are shown having some texture 25, i.e., not a matte 
image. As shown in the test image after processing the texture 25 has been shifted horizontally and/or vertically. An 
arbitrary test region or block 28 is formed, shown in this example as a rectangle, having a reference point Xo, Yo and 
5 is overlaid on both the reference and test images. For a rectangle as shown the reference point and horizontal and 
vertical extents AX, AY define the test region or block28. The test region 28 is located where there is substantial 
texture 25 in the images. 

[0013] A high precision sub-pixel spatial alignment algorithm is shown in Fig. 3. An initialization 
module 30 provides the corresponding reference and test images together with a test block to a correlation 
10 measurement module 32. The first step 34 in the initialization module 30 controls the video capture module 16 to 
capture corresponding reference and test images or frames from the reference and test video signals. The second 
step 36 establishes a test block for overlaying on the respective images in an area that has significant texture 25. The 
first step 38 of the correlation measurement module 32 applies a fast Fourier transform (FFT) to the pixels of the 
images that lie within the test region. A cross-correlation step 40 is then performed in the FFT domain. 

FFT(corr) = FFT(ref)*FFT(tst) 

10 FFT" 1 a(corr) produces correlation coefficients for every shift point within the test region, which may be 
represented in the form of a surface 42 as shown in Fig. 4. The position of a peak 44 in the surface 42 indicates the 
amount of shift in position between the reference and test images. A curve-fit step 46 provides a nearest integer pixel 
shift position for the peak44 based upon the coefficients for the peak position and the positions up, down, left and 
right from the peak position. The pixel shift position from the correlation measure module 32 is input to an update 

20 shift position step 48. For the first iteration only the nearest integer pixel shift position is used. 

[0014] A testing step 50 checks the amount of position shift determined from the update shift position step 48 and 
the number of iterations against specified parameters to determine whether the spatial alignment algorithm is done. 
If the number of iterations is equal to the specified number, or the amount of update is less than a specified noise 
value, then the shift position value from the update step 48 is provided as an output pixel shift position and the 
algorithm ends. Otherwise a shift test block step 52 shifts the test region in the test image by the value from the 

25 update position step48 so that the reference point is shifted to be X+X A , Y+Y A , where X A and Y A are the integer pixel 
shift position values. With the new position of the test region or block, the correlation module 32 is applied again to 
obtain a fractional pixel shift value in X and Y. In the FFT step 38 only the test image is processed after the initial 
iteration. The FFT for the reference image does not change. The fractional pixel shift value is used to update the 
shift position in the update step 48 so that the shift position becomes Xo+X A +X,, Yo+Y A +Y f . It has been found that for 
most practical applications two fractional pixel shift position iterations produce satisfactory results. The final 
shift position value output from the spatial alignment detect module 18 is then used to spatially align the test and 
reference images as indicated above. 

[0015] Thus the present invention provides high precision sub-pixel spatial alignment of digital images by using 
an iterative process to cross-correlate FFTs for a test block from each image in order to find a peak representative 
of a shift position between the images, then by fitting a curve to the correlation coefficients to determine a 
35 nearest integer pixel shift position, updating the position of the test block in the test image and repeating until 
a specified number of iterations occurs or a change in fractional pixel shift position is less than a noise value. 



Claims 

1. A method of high precision sub-pixel spatial alignment of digital images comprising the steps of: 

40 

capturing a reference image from a reference video signal as one of the digital images and a corresponding 
test image from a test video signal as another of the digital images, where the test video signal represents 
the reference video signal after processing; 

overlaying an arbitrary test block on the reference and test images at corresponding locations that include 
45 texture; 

performing a fast Fourier transform for the portions of each image that lie within the test block to produce 
a reference FFT and a test FFT; 

cross-correlating the reference and test FFTs to produce a correlation surface having a peak correlation 
50 value, the position of the peak correlation value on the correlation surface being representative of a shift 

position between the reference and test images; 

fitting a curve to the peak and neighboring correlation values to determine an integer pixel shift position; 

55 updating the position of the test block overlaying the test image by the integer pixel shift position; and 

repeating the performing step for the test image, the correlating step for finding fractional pixel shift 
positions, and the updating step by adding the fractional pixel shift positions to the integer pixel shift 
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position until an end condition is achieved, the final position of the test block in the test image from the 
updating step relative to the initial position of the test block in the test image providing the high 
precision sub-pixel shift alignment between the reference and test images. 



A method of high precision sub-pixel spatial alignment of digital images comprising the steps of: 

cross-correlating in the frequency domain a test block for each of the digital images to produce a 
correlation surface having a peak correlation value at a particular position within the test block; 

updating the test block in one of the digital images as a function of the particular position; and 

repeating the cross-correlating and updating steps until a specified end condition is achieved, the final 
position of the test block in the one of the digital images relative to the other digital image representing 
the high precision sub-pixel spatial alignment between the digital images. 
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